attempting to fix racy Akka.Persistence.TestKit.Tests #7068
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Took ~5 seconds to complete recovery with the Akka.Persistence.TestKit journal? Something's off then. |
Aaronontheweb
commented
Jan 19, 2024
| try | ||
| { | ||
| var highSequenceNr = await _breaker.WithCircuitBreaker((message, readHighestSequenceNrFrom, awj: this), state => | ||
| state.awj.ReadHighestSequenceNrAsync(state.message.PersistenceId, state.readHighestSequenceNrFrom)); | ||
| state.awj.ReadHighestSequenceNrAsync(state.message.PersistenceId, state.readHighestSequenceNrFrom)) | ||
| .ConfigureAwait(false); |
There was a problem hiding this comment.
Wondering if there's some issues with the Akka.Persistence Journal capturing context here, since these tasks run as detatched. I doubt it's the issue but it's one of the things that stood out to me here.
There was a problem hiding this comment.
The task isn't detached though, its actually awaited inside the circuit breaker, except when the circuit breaker is open, in which it just throws an open circuit exception.
There was a problem hiding this comment.
That's true - we have some other racy CircuitBreaker specs too (independently from any Akka.Persistence plugin) - wonder if that's the common issue?
There was a problem hiding this comment.
Its not the CircuitBreaker, the giant lag on recovery is between PreStart and when the LoadSnapshot message was received by the snapshot store actor:
[INFO][02/01/2024 18:30:23.919Z][Thread 0017][PersistenceExtension (akka://test)] Auto-starting journal plugin `akka.persistence.journal.test`
[INFO][02/01/2024 18:30:23.940Z][Thread 0017][PersistenceExtension (akka://test)] Auto-starting snapshot store `akka.persistence.snapshot-store.test`
[INFO][02/01/2024 18:30:23.999Z][Thread 0005][akka://test/system/akka.persistence.journal.test] Using write interceptor Noop
[INFO][02/01/2024 18:30:24.041Z][Thread 0012][akka://test/user/$b] Starting up and beginning recovery
[INFO][02/01/2024 18:30:29.807Z][Thread 0012][akka://test/system/akka.persistence.snapshot-store.test] LoadSnapshot message received.
[INFO][02/01/2024 18:30:29.809Z][Thread 0012][akka://test/system/akka.persistence.snapshot-store.test] Starting LoadSnapshotAsync circuit breaker.
[INFO][02/01/2024 18:30:29.810Z][Thread 0012][akka://test/system/akka.persistence.snapshot-store.test] Invoking LoadAsync inside circuit breaker.
[INFO][02/01/2024 18:30:29.812Z][Thread 0012][akka://test/system/akka.persistence.snapshot-store.test] Starting to intercept snapshot foo loading using interceptor Noop
[INFO][02/01/2024 18:30:29.812Z][Thread 0012][akka://test/system/akka.persistence.snapshot-store.test] Completed intercept snapshot foo loading using interceptor Noop
[INFO][02/01/2024 18:30:29.814Z][Thread 0012][akka://test/system/akka.persistence.snapshot-store.test] LoadSnapshotAsync circuit breaker completed.
[DEBUG][02/01/2024 18:30:29.838Z][Thread 0012][akka://test/system/akka.persistence.journal.test/$a] Replay completed: RecoverySuccess<highestSequenceNr: 0>
[INFO][02/01/2024 18:30:29.839Z][Thread 0012][akka://test/user/$b] Received recover Akka.Persistence.RecoveryCompleted
There was a problem hiding this comment.
This makes me think that the bug is actually in the RecoveryPermitter class
This reverts commit 30897ff.
|
Well this is puzzling, but reflects the problem using logging statements: |
|
When that same test passes, the logs look like this: |
|
|
|
So what is blocking the |
…h so that it wouldn't be masked by other exceptions
…heweb/akka.net into fix-AkkaPersistenceTestKitTests
Arkatufus
added a commit
to Arkatufus/akka.net
that referenced
this pull request
Feb 1, 2024
4 tasks
That is a red herring, something is blocking the persistent actor under test thread right before recovery even started. These are logs taken from 2 different tests, one is from Bug4762FixSpecCounterActorTestsObservations
|
|
Note that there are no async-await between the two log lines, only a simple code. The The |
|
Changing the actor dispatcher to "akka.actor.internal-dispatcher" eliminates the start-up delay, could it be that there's something wrong inside "akka.actor.default-dispatcher" implementation? |
We implemented this in #7117 - let's see how it does |
|
@Arkatufus looks to me like the dispatcher changes didn't have any impact |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Trying to fix a pair of specs with high flip rates in the Akka.NET test suite.
Checklist
For significant changes, please ensure that the following have been completed (delete if not relevant):